Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix for some AWs that do not complete #658

Merged
merged 5 commits into from
Oct 11, 2023
Merged

fix for some AWs that do not complete #658

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
09735b4
fix for some AWs that do not complete
asm582 Oct 10, 2023
897e1ba
fix AW to make resources consistent
asm582 Oct 10, 2023
2fc46ba
check AW status instead of jobs
asm582 Oct 10, 2023
e66a7eb
changing jobstatus from jobs to appwrappers
jbusche Oct 11, 2023
10b5a78
Merge pull request #1 from jbusche/adjust_perf
asm582 Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion pkg/controller/queuejob/queuejob_controller_ex.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1625,7 +1625,10 @@ func (cc *XController) addQueueJob(obj interface{}) {
}
if latestAw.Status.State != arbv1.AppWrapperStateActive && latestAw.Status.State != arbv1.AppWrapperStateEnqueued && latestAw.Status.State != arbv1.AppWrapperStateRunningHoldCompletion {
klog.V(2).Infof("[Informer-addQJ] Stopping requeue for AW %s/%s with status %s", latestAw.Namespace, latestAw.Name, latestAw.Status.State)
break // Exit the loop
AwinEtcd, err := cc.arbclients.WorkloadV1beta1().AppWrappers(latestAw.Namespace).Get(context.Background(), latestAw.Name, metav1.GetOptions{})
if AwinEtcd.Status.State == latestAw.Status.State && err != nil {
break // Exit the loop
}
}
// Enqueue the latest copy of the AW.
if (qj.Status.State != arbv1.AppWrapperStateCompleted && qj.Status.State != arbv1.AppWrapperStateFailed) && hasCompletionStatus {
Expand Down
4 changes: 2 additions & 2 deletions test/perf-test/perf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ check_kubectl_login_status
# Track whether you have the MCAD controller installed
echo "Checking MCAD Controller installation status"
echo
check_mcad_installed_status
#check_mcad_installed_status

echo
read -p "How many appwrapper jobs do you want?" jobs
Expand Down Expand Up @@ -114,7 +114,7 @@ done
esac

# Check for all jobs to report complete
jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=1 |wc -l`
jobstatus=`kubectl get appwrappers -n default --no-headers --field-selector status.successful=1 |wc -l`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shoot - this change isn't working right - getting the error message:

Error from server (BadRequest): Unable to find "workload.codeflare.dev/v1beta1, Resource=appwrappers" that match label selector "", field selector "status.successful=1": field label not supported: status.successful

But since the jobstatus later in the while loop uses jobs the perf.sh script still completes.


while [ $jobstatus -lt $jobs ]
do
Expand Down
13 changes: 6 additions & 7 deletions test/perf-test/preempt-exp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,14 @@ spec:
growthType: "exponential"
priority: 9
resources:
Items: []
GenericItems:
- replicas: 1
completionstatus: Complete
custompodresources:
- replicas: 1
requests:
cpu: 10m
memory: 10M
cpu: 500m
memory: 128M
nvidia.com/gpu: 0
limits:
cpu: 500m
Expand All @@ -31,8 +30,8 @@ spec:
metadata:
namespace: default
name: defaultaw-schd-spec-with-timeout-1
# labels:
# appwrapper.mcad.ibm.com: defaultaw-schd-spec-with-timeout-1
labels:
appwrapper.mcad.ibm.com: defaultaw-schd-spec-with-timeout-1
spec:
parallelism: 1
completions: 1
Expand All @@ -49,8 +48,8 @@ spec:
args: [ "sleep 10" ]
resources:
requests:
memory: "10Mi"
cpu: "10m"
memory: "128Mi"
cpu: "500m"
limits:
memory: "128Mi"
cpu: "500m"
Expand Down