Class LabeledCriteriaEvalChain

constructor

• new LabeledCriteriaEvalChain(fields): LabeledCriteriaEvalChain

• Parameters

  • fields: LLMChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>

  Returns LabeledCriteriaEvalChain

llm

outputKey

outputParser

prompt

requiresInput

skipReferenceWarning

Optional criterionName

Optional evaluationName

Optional llmKwargs

Optional memory

Optional skipInputWarning

inputKeys

• get inputKeys(): string[]

• Returns string[]

outputKeys

• get outputKeys(): string[]

• Returns string[]

apply

• apply(inputs, config?): Promise<ChainValues[]>

• Parameters

  • inputs: ChainValues[]

  • Optional config: (RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[])[]

  Returns Promise<ChainValues[]>

  ⚠️ Deprecated ⚠️

  Use .batch() instead. Will be removed in 0.2.0.

  This feature is deprecated and will be removed in the future.

  It is not recommended for use.

  Call the chain on all inputs in the list

call

• call(values, config?): Promise<ChainValues>

• Run the core logic of this chain and add to output if desired.

  Wraps _call and handles memory.

  Parameters

  • values: any

  • Optional config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

  Returns Promise<ChainValues>

checkEvaluationArgs

• checkEvaluationArgs(reference?, input?): void

• Check if the evaluation arguments are valid.

  Parameters

  • Optional reference: string

    The reference label.

  • Optional input: string

    The input string.

  Returns void

  Throws

  If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided.

evaluateStrings

• evaluateStrings(args, config?): Promise<ChainValues>

• Evaluate Chain or LLM output, based on optional input and label.

  Parameters

  • args: StringEvaluatorArgs & BaseLanguageModelCallOptions

  • Optional config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

  Returns Promise<ChainValues>

  The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys:

  • score: the score of the evaluation, if applicable.
  • value: the string value of the evaluation, if applicable.
  • reasoning: the reasoning for the evaluation, if applicable.

getEvalInput

• getEvalInput(__namedParameters): CriteriaEvalInput

• Parameters

  • __namedParameters: StringEvaluatorArgs

  Returns CriteriaEvalInput

invoke

• invoke(input, options?): Promise<ChainValues>

• Invoke the chain with the provided input and returns the output.

  Parameters

  • input: ChainValues

    Input values for the chain run.

  • Optional options: RunnableConfig

  Returns Promise<ChainValues>

  Promise that resolves with the output of the chain run.

predict

• predict(values, callbackManager?): Promise<EvalOutputType>

• Format prompt with values and pass to LLM

  Parameters

  • values: any

    keys to pass to prompt template

  • Optional callbackManager: CallbackManager

    CallbackManager to use

  Returns Promise<EvalOutputType>

  Completion from LLM.

  Example

  llm.predict({ adjective: "funny" })
  Copy

prepOutputs

• prepOutputs(inputs, outputs, returnOnlyOutputs?): Promise<Record<string, unknown>>

• Parameters

  • inputs: Record<string, unknown>

  • outputs: Record<string, unknown>

  • returnOnlyOutputs: boolean = false

  Returns Promise<Record<string, unknown>>

run

• run(input, config?): Promise<string>

• Parameters

  • input: any

  • Optional config: RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

  Returns Promise<string>

  Deprecated

  Use .invoke() instead. Will be removed in 0.2.0.

serialize

• serialize(): SerializedLLMChain

• Returns SerializedLLMChain

  Deprecated

Static deserialize

• deserialize(data): Promise<LLMChain<string, BaseLanguageModel<any, BaseLanguageModelCallOptions>>>

• Parameters

  • data: SerializedLLMChain

  Returns Promise<LLMChain<string, BaseLanguageModel<any, BaseLanguageModelCallOptions>>>

Static fromLLM

• fromLLM(llm, criteria?, chainOptions?): Promise<CriteriaEvalChain>

• Create a new instance of the CriteriaEvalChain.

  Parameters

  • llm: BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>

  • Optional criteria: "detail" | ConstitutionalPrinciple | {
        [key: string]: string;
    } | "conciseness" | "relevance" | "correctness" | "coherence" | "harmfulness" | "maliciousness" | "helpfulness" | "controversiality" | "misogyny" | "criminality" | "insensitivity" | "depth" | "creativity"

  • Optional chainOptions: Partial<Omit<LLMEvalChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>, "llm">>

    Options to pass to the constructor of the LLMChain.

  Returns Promise<CriteriaEvalChain>

Static resolveCriteria

• resolveCriteria(criteria?): Record<string, string>

• Resolve the criteria to evaluate.

  Parameters

  • Optional criteria: "detail" | ConstitutionalPrinciple | {
        [key: string]: string;
    } | "conciseness" | "relevance" | "correctness" | "coherence" | "harmfulness" | "maliciousness" | "helpfulness" | "controversiality" | "misogyny" | "criminality" | "insensitivity" | "depth" | "creativity"

    The criteria to evaluate the runs against. It can be: - a mapping of a criterion name to its description - a single criterion name present in one of the default criteria - a single ConstitutionalPrinciple instance

  Returns Record<string, string>

  A dictionary mapping criterion names to descriptions.