qgyd2021/lip_service_4chan

lib_service_4chan

This model is a fine-tuned version of uer/gpt2-chinese-cluecorpussmall on the lip_service_4chan dataset.

Lip Service 满嘴芬芳，吵架陪练员。

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.716	0.01	100	1.9495
1.8985	0.02	200	1.6915
1.7151	0.02	300	1.5763
1.6217	0.03	400	1.5115
1.564	0.04	500	1.4694
1.5461	0.05	600	1.4379
1.4943	0.06	700	1.4127
1.4737	0.07	800	1.3890
1.4399	0.07	900	1.3813
1.4356	0.08	1000	1.3540
1.3999	0.09	1100	1.3329
1.3668	0.1	1200	1.3153
1.3604	0.11	1300	1.3029
1.3352	0.12	1400	1.2834
1.3278	0.12	1500	1.2619
1.315	0.13	1600	1.2539
1.2854	0.14	1700	1.2432
1.292	0.15	1800	1.2288
1.2795	0.16	1900	1.2188
1.2677	0.16	2000	1.2059
1.2599	0.17	2100	1.2019
1.2479	0.18	2200	1.1915
1.2245	0.19	2300	1.1827
1.2326	0.2	2400	1.1734
1.2124	0.21	2500	1.1660
1.2171	0.21	2600	1.1576
1.1917	0.22	2700	1.1518
1.1867	0.23	2800	1.1444
1.1821	0.24	2900	1.1386
1.1741	0.25	3000	1.1347
1.1753	0.25	3100	1.1293
1.1629	0.26	3200	1.1264
1.1694	0.27	3300	1.1201
1.1482	0.28	3400	1.1146
1.156	0.29	3500	1.1052
1.1512	0.3	3600	1.0982
1.142	0.3	3700	1.0971
1.1544	0.31	3800	1.0920
1.1312	0.32	3900	1.0869
1.1394	0.33	4000	1.0808
1.123	0.34	4100	1.0747
1.1154	0.35	4200	1.0715
1.1064	0.35	4300	1.0674
1.1245	0.36	4400	1.0620
1.1036	0.37	4500	1.0575
1.0963	0.38	4600	1.0568
1.0987	0.39	4700	1.0491
1.0859	0.39	4800	1.0443
1.0845	0.4	4900	1.0432
1.0938	0.41	5000	1.0410
1.087	0.42	5100	1.0334
1.077	0.43	5200	1.0324
1.0787	0.44	5300	1.0276
1.068	0.44	5400	1.0220
1.0748	0.45	5500	1.0199
1.0622	0.46	5600	1.0169
1.0555	0.47	5700	1.0153
1.0498	0.48	5800	1.0100
1.055	0.49	5900	1.0074
1.0424	0.49	6000	1.0020
1.0465	0.5	6100	0.9976
1.0414	0.51	6200	0.9942
1.0355	0.52	6300	0.9919
1.0234	0.53	6400	0.9883
1.0205	0.53	6500	0.9857
1.0316	0.54	6600	0.9805
1.0137	0.55	6700	0.9788
1.0222	0.56	6800	0.9773
1.0219	0.57	6900	0.9722
1.0032	0.58	7000	0.9706
1.0039	0.58	7100	0.9669
1.0166	0.59	7200	0.9635
1.0065	0.6	7300	0.9614
1.0087	0.61	7400	0.9574
0.9968	0.62	7500	0.9525
1.0031	0.62	7600	0.9503
0.99	0.63	7700	0.9491
0.9946	0.64	7800	0.9457
0.9944	0.65	7900	0.9424
0.9854	0.66	8000	0.9399
0.9797	0.67	8100	0.9364
0.9804	0.67	8200	0.9341
0.9835	0.68	8300	0.9318
0.9849	0.69	8400	0.9299
0.9753	0.7	8500	0.9274
0.975	0.71	8600	0.9238
0.9649	0.72	8700	0.9225
0.9654	0.72	8800	0.9202
0.958	0.73	8900	0.9167
0.9679	0.74	9000	0.9143
0.9631	0.75	9100	0.9110
0.9633	0.76	9200	0.9086
0.9495	0.76	9300	0.9071
0.9625	0.77	9400	0.9036
0.9519	0.78	9500	0.9023
0.9399	0.79	9600	0.8993
0.9624	0.8	9700	0.8973
0.9418	0.81	9800	0.8963
0.9394	0.81	9900	0.8933
0.947	0.82	10000	0.8919
0.9326	0.83	10100	0.8900
0.9326	0.84	10200	0.8886
0.9343	0.85	10300	0.8860
0.9263	0.85	10400	0.8841
0.9256	0.86	10500	0.8818
0.9373	0.87	10600	0.8807
0.9314	0.88	10700	0.8789
0.9203	0.89	10800	0.8770
0.927	0.9	10900	0.8754
0.934	0.9	11000	0.8744
0.9193	0.91	11100	0.8727
0.9185	0.92	11200	0.8714
0.9188	0.93	11300	0.8702
0.9165	0.94	11400	0.8693
0.9209	0.95	11500	0.8682
0.9241	0.95	11600	0.8670
0.9182	0.96	11700	0.8662
0.9076	0.97	11800	0.8653
0.9225	0.98	11900	0.8643
0.9094	0.99	12000	0.8640
0.913	0.99	12100	0.8635

Framework versions

• Transformers 4.33.0
• Pytorch 2.0.0
• Datasets 2.1.0
• Tokenizers 0.13.3