Data quality: control, termination, and validation¶

Copy of sumaris-app/.agents/rules/service-quality.md (rule written for AI coding agents, when implementing data quality control, termination, or validation in services). Do not edit here: run npm run site:sync-rules to update it.

Services managing root data entities implement a quality lifecycle with three main phases: control (client-side validation), termination (marking as controlled/ready), and validation (server-side approval).

Interfaces¶

IDataEntityQualityService<T>: defines control(), qualify(), and canUserWrite() for any data entity (including child entities like Operation, Batch);
IRootDataEntityQualityService<T>: extends the above with terminate(), validate(), and unvalidate() for root entities (e.g. Trip, ObservedLocation);

Control¶

The control() method validates an entity using the validator service and, for parent entities, delegates control to child services recursively.

Pattern for a root entity (e.g. `TripService.control()`)¶

async control(entity: Trip, opts?: TripControlOptions): Promise<AppErrorWithDetails> {
  // 1. Load program and strategy
  const program = opts?.program ?? (await this.programRefService.loadByLabel(entity.program.label));
  const strategy = opts?.strategy ?? (await this.loadStrategy(program, entity));

  // 2. Validate the entity itself using the validator service
  const form = this.validatorService.getFormGroup(entity, {
    ...opts,
    program,
    isOnFieldMode: false,
    withMeasurements: true,
  });

  if (!form.valid) {
    await AppFormUtils.waitWhilePending(form);
    if (form.invalid) {
      const errors = AppFormUtils.getFormErrors(form);
      return { message: 'COMMON.FORM.HAS_ERROR', details: { errors } };
    }
  }

  // 3. Delegate control to child services
  // Control physical gears
  const gearErrors = await this.physicalGearService.controlAllByTrip(entity, { program, strategy, ... });
  if (gearErrors) return { message: 'TRIP.ERROR.INVALID_GEARS', details: { errors: { gears: gearErrors } } };

  // Control operations
  const opErrors = await this.operationService.controlAllByTrip(entity, { program, strategy, ... });
  if (opErrors) return { message: 'TRIP.ERROR.INVALID_OPERATIONS', details: { errors: { operations: opErrors } } };

  return undefined; // No errors
}

Pattern for a child entity (e.g. `OperationService.control()`)¶

async control(entity: Operation, opts?: OperationControlOptions): Promise<FormErrors> {
  // 1. Validate the entity itself
  const form = this.validatorService.getFormGroup(entity, opts);
  if (!form.valid) {
    await AppFormUtils.waitWhilePending(form);
    if (form.invalid) return AppFormUtils.getFormErrors(form);
  }

  // 2. Delegate to deeper child services (e.g. BatchService)
  if (entity.catchBatch) {
    const errors = await this.batchService.control(entity.catchBatch, { program: opts.program, ... });
    if (errors) return { catch: { invalidOrIncomplete: true } };
  }

  // 3. Mark as controlled if local entity
  if (EntityUtils.isLocalId(entity.id) && DataEntityUtils.isNotControlled(entity)) {
    DataEntityUtils.markAsControlled(entity);
    await this.save(entity);
  }

  return undefined;
}

`controlAllByTrip()` pattern¶

Child services expose a controlAllByTrip() method that iterates over all children of a parent entity, controls each one, and either terminates it (if valid) or marks it as invalid:

async controlAllByTrip(trip: Trip, opts?: OperationControlOptions): Promise<FormErrors> {
  const { data } = await this.loadAllByTrip({ tripId: trip.id }, { fullLoad: false });
  if (isEmptyArray(data)) return undefined;

  let errorsById: FormErrors = null;

  for (let entity of data) {
    entity = await this.load(entity.id); // Load full entity

    const errors = await this.control(entity, opts);

    if (errors) {
      errorsById = errorsById || {};
      errorsById[entity.id] = errors;

      // Translate error and mark entity as invalid
      const errorMessage = this.formErrorTranslator.translateErrors(errors, opts.translatorOptions);
      DataEntityUtils.markAsInvalid(entity, errorMessage);
      await this.save(entity);
    } else {
      // Control passed: terminate the entity
      await this.terminate(entity);
    }
  }

  return errorsById;
}

Termination¶

The terminate() method marks an entity as controlled and ready for validation.

For root entities (in `RootDataSynchroService`)¶

Local entities: marks as READY_TO_SYNC via RootDataEntityUtils.markAsReadyToSync(), then saves locally;
Remote entities: calls a GraphQL terminate mutation on the server, which sets controlDate;

async terminate(entity: T, opts?: TO): Promise<T> {
  if (EntityUtils.isLocal(entity)) {
    await this.fillOfflineDefaultProperties(entity);
    RootDataEntityUtils.markAsReadyToSync(entity);
    const json = this.asObject(entity, MINIFY_DATA_ENTITY_FOR_LOCAL_STORAGE);
    await this.entities.save(json);
    return entity;
  }
  // Remote: delegate to base class (GraphQL mutation)
  return super.terminate(entity, opts);
}

For child entities (e.g. `OperationService.terminate()`)¶

Cleans qualificationComments;
Handles special quality flags (e.g. abnormal operations);
Saves locally if the parent trip is local, or calls a GraphQL terminate mutation;

Validation¶

The validate() method is called after termination, typically by a supervisor or administrator. It requires:

The entity to be saved remotely (id must be a remote ID);
The entity to be already controlled (controlDate must be set);
The entity to not be already validated (validationDate must be null);

async validate(entity: T, opts?: VO): Promise<T> {
  if (isNil(entity.controlDate)) throw new Error('Entity must be controlled before validate!');
  if (isNotNil(entity.validationDate)) throw new Error('Entity is already validated!');

  const json = this.asObject(entity, SAVE_AS_OBJECT_OPTIONS);
  await this.graphql.mutate<{ data: T }>({
    mutation: this.mutations.validate,
    variables: { data: json, options: opts?.withChildren ? { withChildren: true } : undefined },
    error: { code: DataErrorCodes.VALIDATE_ENTITY_ERROR, message: 'ERROR.VALIDATE_ENTITY_ERROR' },
    update: (cache, { data }) => {
      this.copyIdAndUpdateDate(data?.data as T, entity);
    },
  });
  return entity;
}

The unvalidate() method reverses validation, requiring the entity to already be validated.

Progression tracking¶

All control methods support progression tracking via IProgressionOptions:

progression: an IProgressionModel instance shared across the control chain;
maxProgression: the maximum progression value allocated to this step;
Each level divides its maxProgression among its sub-steps and calls progression.increment();

Error handling¶

control() returns AppErrorWithDetails (root entities) or FormErrors (child entities) on failure, undefined on success;
DataEntityUtils.markAsInvalid(entity, errorMessage) stores the error in qualificationComments and sets the quality flag;
DataEntityUtils.markAsControlled(entity) sets controlDate and clears errors;
Error messages are translated via FormErrorTranslator.translateErrors() before being stored;

Delegation chain example¶

For a Trip:

TripService.control() validates the trip form;
Delegates to PhysicalGearService.controlAllByTrip() for gear validation;
Delegates to OperationService.controlAllByTrip() which iterates operations;
For each operation, OperationService.control() validates the operation form;
Delegates to BatchService.control() for catch batch validation;
If all pass, each operation is terminated via OperationService.terminate();

GraphQL mutations¶

Services declare terminate, validate, unvalidate, and qualify mutations in their Mutations constant:

const Mutations: BaseRootEntityGraphqlMutations = {
  save: gql`...`,
  deleteAll: gql`...`,
  terminate: gql`mutation TerminateTrip($data: TripVOInput!) { data: terminateTrip(trip: $data) { ...TripFragment } }`,
  validate: gql`mutation ValidateTrip($data: TripVOInput!) { data: validateTrip(trip: $data) { ...TripFragment } }`,
  unvalidate: gql`mutation UnvalidateTrip($data: TripVOInput!) { data: unvalidateTrip(trip: $data) { ...TripFragment } }`,
  qualify: gql`mutation QualifyTrip($data: TripVOInput!) { data: qualifyTrip(trip: $data) { ...TripFragment } }`,
};

Reference examples¶

Root entity quality service: src/app/trip/trip/trip.service.ts (control() method);
Child entity quality service: src/app/trip/operation/operation.service.ts (control(), controlAllByTrip(), terminate());
Leaf entity control: src/app/trip/batch/common/batch.service.ts (control());
Quality service interfaces: src/app/data/services/data-quality-service.class.ts;
Base terminate/validate: src/app/data/services/root-data-service.class.ts;
Offline terminate: src/app/data/services/root-data-synchro-service.class.ts;