Implement an LLM-as-a-judge. Its task is to judge the outputs of other LLMs. Your judge will be presented with four completions to a given prompt. It needs to judge which one is the best. Use Gemini 2.0 Flash and try to achieve a high accuracy on the provided dataset. ## Requirements 1. Install dependencies: `pip install -r requirements.txt` (tested with Python 3.10). 2. Copy `.env_example` to `.env` and fill in the API key we provided. 3. Implement your full solution in `main.py` only. Do not add any other files. They will be ignored. 4. Before submitting, run `python run_submission.py --debug` to check if your solution is gradable and bug-free. - If you want to check the accuracy on the full dataset, omit the `--debug` flag. - Make sure that your solution runs in under 20 minutes on the full dataset. - If you run into rate limits you can lower `CONCURRENCY` in `run_submission.py`. Please leave the `requirements.txt` unchanged and don't add any other dependencies. You can request usage information for the provided API key from [OpenRouter](https://openrouter.ai/docs/api-reference/api-keys/get-current-api-key). The provided API key will be deactivated after the challenge. ## Submission - Create a private GitHub repository. - If you're Github repo is public you will spoil the challenge for others. We will automatically reject your submission. - Please name the repository `ellamind_coding_challenge_John_Doe` where `John_Doe` is your name. - Invite `ellamind-admin` as collaborator to your repository. - Ensure your solution is present on the main branch. - No further changes will be accepted after the two hour time limit. - If you don't follow these instructions the grading will fail. ## Code Storage and Analysis Your submitted code will be stored and analyzed by ellamind for the purpose of evaluating your application and improving our recruitment process. ## Additional Notes - Grading will use a private test-set with the same structure. - Perfect results aren't expected.