Regardless of their exceptional achievements, fashionable Giant Language Fashions (LLMs) encounter exorbitant computational and reminiscence footprints. Not too long ago, a number of works have proven important success in training-free and data-free compression (pruning and quantization) of LLMs reaching 50-60% sparsity and decreasing the bit-width down to three or 4 bits per weight, with negligible perplexity degradation over the uncompressed baseline. As current analysis efforts are targeted on creating more and more refined compression strategies, our work takes a step again, and re-evaluates the effectiveness of current SoTA compression strategies, which depend on a reasonably easy and extensively questioned metric, perplexity (even for dense LLMs). We introduce Data-Intensive Compressed LLM BenchmarK (LLM-KICK), a set of carefully-curated duties to re-define the analysis protocol for compressed LLMs, which have important alignment with their dense counterparts, and perplexity fail to seize refined change of their true capabilities. LLM-KICK unveils many favorable deserves and unlucky plights of present SoTA compression strategies: all pruning strategies endure important efficiency degradation, generally at trivial sparsity ratios (e.g., 25-30%), and fail for N:M sparsity on knowledge-intensive duties; present quantization strategies are extra profitable than pruning; but, pruned LLMs even at 50% sparsity are sturdy in-context retrieval and summarization programs; amongst others. LLM-KICK is designed to holistically entry compressed LLMs’ capability for language understanding, reasoning, era, in-context retrieval, in-context summarization, and so forth. We hope our examine can foster the event of higher LLM compression strategies.