The How of the Data Matters
Apple internship paper discloses reformatting the training data for pre-training yields better results
This Apple internship paper discloses a now classic technique: reformatting the training data for pre-training yields better results. Namely, in this case, rephrasing the queries as Wikipedia entries:
