Skip to content

14. Open questions

  • The npm scope @trialkit: register the org before the first publish.
  • The Node.js floor: keep it in every package, or set each package’s floor from what its own contract and smoke legs prove (§5)?
  • MLflow evaluation datasets (Unity Catalog tables on Databricks) as a case source: in report-mlflow, or a separate dataset-mlflow package?
  • Idempotent MLflow upload: uploadJsonl does not check whether MLflow already has a run, so a file that you send twice makes two runs. The tag trialkit.run_id would let the uploader find and skip a run, at one runs/search request for each run and with the permission to search. Always, behind an option, or never?
  • Python runner adapters after pytest: none, or unittest?
  • The license terms for templates that copy another tool’s prompt.
  • run records the output of every task call, so a history in a Git branch grows with it. Should a reporter leave out the outputs of trials that passed?