Prevalence and Prevention of LLM

Prevalence and prevention of large language model use in crowd work

Summary

Veselovsky et al. (2023) have conducted a study regarding prevalence and prevention of LLM (large language model) use in crowd work. Crowd work platforms are those that distribute works to individuals, like Prolific and Amazon Mechanical Turk. In our case, the conjoint survey is a crowd work as well.

Their primary findings are: without any intentional direction, the estimated prevalence of LLM use was 30%. However, if there are literal limits or copy-pasting limits, the usage was decreased by a half. Their secondary analyses are: LLM produces high-quality but homogeneous responses, which may harm future LLM training.

Results

  1. Hurdles and requests both reduce the usage of LLM.
  2. LLM using is not equal to cheating. LLM can assist crowd workers.
  3. Stricter mitigation approaches may backfire.
  4. LLM using can only be prevented to a certain degree, but cannot be eliminated.
  5. This study also uses a logit equation to scale the temperature of pre-trained model.

Gaps

  1. This paper has raised a good topic but lacks a solid reasoning. The 2 studies are both conducted in July 2023 with interval of 3 weeks, and the sampling and data analysis strategies look arbitrary.
  2. The discussion part only contains classifying tables and a dumbbell chart. I am expecting a more in-depth interpretation but there wasn’t.
  3. The tables and dumbbell chart are all too information-crowded. It takes a long time to read, digest, and summarize what they intend to conclude.
Back to top

References

Veselovsky, Veniamin, Manoel Horta Ribeiro, Philip Cozzolino, Andrew Gordon, David Rothschild, and Robert West. 2023. “Prevalence and Prevention of Large Language Model Use in Crowd Work.” https://doi.org/10.48550/ARXIV.2310.15683.