Pandas: jadval ma'lumotlari bilan ishlash

ML loyihasida vaqtning 60–80% i modelga emas, ma'lumotga ketadi: yuklash, tushunish, tozalash, birlashtirish, xususiyatlar (features) yaratish. Bu ishning asosiy vositasi — Pandas. U NumPy ustiga qurilgan va jadval ma'lumotlari uchun Excel + SQL + Python'ning birlashmasiga o'xshaydi. LLM loyihalarida ham doim kerak: eval natijalarini tahlil qilish, datasetlarni tayyorlash, xarajatlarni hisoblash.

uv add pandas pyarrow      # pyarrow — Parquet va tezkor satrlar uchun

import numpy as np
import pandas as pd

Python Pandas Tutorial (Part 1): Getting Started with Data Analysis - Installation and Loading Data
Pandas seriyasining birinchi qismi: o'rnatish, ma'lumot yuklash va DataFrame bilan tanishish. Seriyaning keyingi qismlari (filtrlash, groupby, tozalash) bobdagi mavzularni to'liq qoplaydi.
YouTube'da ochish ↗

1. Mashq uchun dataset#

Butun bob davomida bitta sun'iy dataset bilan ishlaymiz — onlayn do'kon buyurtmalari. Uni o'zingiz yaratib, har bir misolni takrorlay olasiz:

rng = np.random.default_rng(7)
n = 1000
orders = pd.DataFrame({
    "order_id": np.arange(1, n + 1),
    "date": pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 270, n), unit="D"),
    "city": rng.choice(["Toshkent", "Samarqand", "Buxoro", "Andijon"], n, p=[0.5, 0.2, 0.15, 0.15]),
    "category": rng.choice(["elektronika", "kiyim", "kitob"], n, p=[0.3, 0.45, 0.25]),
    "price": rng.lognormal(mean=12, sigma=0.8, size=n).round(-3),
    "qty": rng.integers(1, 5, n),
    "rating": rng.choice([1, 2, 3, 4, 5, np.nan], n, p=[0.05, 0.05, 0.15, 0.3, 0.3, 0.15]),
})
orders.to_csv("orders.csv", index=False)

2. Series va DataFrame#

Series — bir o'lchamli, nomlangan massiv + indeks (yorliqlar).
DataFrame — Series'lar jadvali: har ustun — Series, hamma ustunlar bitta indeksni baham ko'radi.

s = pd.Series([12000, 15000, 22000], index=["olma", "nok", "uzum"], name="narx")
s["nok"]          # 15000
s.mean()          # 16333.33
s[s > 13000]      # nok, uzum — NumPy'dagi kabi mask
 
df = pd.DataFrame({"meva": ["olma", "nok"], "narx": [12000, 15000]})
df["narx"]        # Series
df[["meva"]]      # DataFrame (ikki qavs!)

Har bir ustun ichida — NumPy massivi (yoki Arrow massivi). Shuning uchun 19-bobdagi hamma narsa — vektorlashtirish, broadcasting, maskalar — Pandas'da ham ishlaydi.

3. O'qish va saqlash#

df = pd.read_csv("orders.csv", parse_dates=["date"])    # sana ustunini darhol datetime qilish
 
pd.read_csv(path, sep=";", encoding="utf-8-sig", usecols=["a", "b"], nrows=1000)
pd.read_parquet("orders.parquet")
pd.read_json("data.jsonl", lines=True)       # JSONL (11-bob)
pd.read_excel("hisobot.xlsx", sheet_name=0)  # uv add openpyxl
pd.read_sql("SELECT * FROM runs", con)       # sqlite3 ulanishi (18-bob)
 
df.to_parquet("orders.parquet", index=False)  # ✅ katta ma'lumot uchun eng yaxshi
df.to_csv("out.csv", index=False)             # index=False — keraksiz indeks ustunisiz
df.to_json("out.jsonl", orient="records", lines=True, force_ascii=False)

Juda katta CSV'ni bo'laklab o'qish (7-bobdagi g'oya):

for chunk in pd.read_csv("huge.csv", chunksize=100_000):
    process(chunk)

4. Ma'lumotni birinchi ko'rish#

Yangi datasetni olganingizda har doim shu buyruqlardan boshlang:

df.head()          # birinchi 5 qator (tail() — oxirgi)
df.shape           # (1000, 7)
df.info()          # ustunlar, tiplar, bo'sh bo'lmagan qiymatlar soni, xotira
df.describe()      # sonli ustunlar statistikasi
df["city"].value_counts()
df.isna().sum()    # har ustunda nechta yetishmayotgan qiymat

   order_id       date      city     category     price  qty  rating
0         1 2026-09-13  Toshkent        kiyim   30000.0    4     3.0
1         2 2026-06-18    Buxoro        kiyim   95000.0    1     5.0
2         3 2026-07-04    Buxoro  elektronika   32000.0    2     4.0
3         4 2026-08-31  Toshkent  elektronika  172000.0    4     5.0
4         5 2026-06-06    Buxoro        kiyim  163000.0    4     5.0

 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   order_id  1000 non-null   int64
 1   date      1000 non-null   datetime64[us]
 2   city      1000 non-null   str
 3   category  1000 non-null   str
 4   price     1000 non-null   float64
 5   qty       1000 non-null   int64
 6   rating    852 non-null    float64

info() darhol ikki narsani ko'rsatdi: ratingda 148 ta qiymat yo'q, va rating — float64 (chunki NaN faqat float'da bo'ladi). describe() esa narxlar taqsimotini: o'rtacha 227 630, mediana 166 000, maksimum 2 094 000 — o'rtacha mediandan ancha katta, demak taqsimot "o'ngga cho'zilgan" (bir nechta juda qimmat buyurtma). Bunday kuzatuvlar — EDA'ning (exploratory data analysis) mohiyati.

5. Tanlash: `[]`, `loc`, `iloc`#

df["price"]                          # bitta ustun → Series
df[["city", "price"]]                # bir nechta ustun → DataFrame
 
df.loc[5]                            # INDEKS YORLIG'I bo'yicha qator
df.loc[df["city"] == "Buxoro", ["order_id", "price", "qty"]]   # loc[qatorlar, ustunlar]
df.loc[0:4, "city"]                  # loc'da oxiri KIRADI (yorliq bo'yicha)
 
df.iloc[0]                           # POZITSIYA bo'yicha
df.iloc[:2, :3]                      # birinchi 2 qator, 3 ustun (oxiri kirmaydi)

	`loc`	`iloc`
Nima bo'yicha	Yorliq (indeks qiymati, ustun nomi)	Pozitsiya (0, 1, 2...)
Slice oxiri	Kiradi	Kirmaydi
Mask	Qabul qiladi	Faqat NumPy massiv
Qachon	Deyarli har doim	"Birinchi n ta" kabi pozitsion ishlar

Filtrlash#

df[df["price"] > 500_000]
df[(df["city"] == "Samarqand") & (df["price"] > 300_000)]     # & va qavslar — NumPy kabi
df[df["city"].isin(["Buxoro", "Andijon"])]
df[df["rating"].between(4, 5)]
df.query("city == 'Samarqand' and price > 300_000")            # SQL'ga o'xshash, o'qilishi oson

6. Yangi ustunlar va vektorlashtirish#

df["revenue"] = df["price"] * df["qty"]                     # vektorlashgan — tez
df["month"] = df["date"].dt.month                           # .dt — sana amallari
df["city_upper"] = df["city"].str.upper()                   # .str — satr amallari
df["is_big"] = df["revenue"] > 1_000_000
df["price_band"] = pd.cut(df["price"],
                          bins=[0, 100_000, 300_000, 1_000_000, np.inf],
                          labels=["arzon", "o'rta", "qimmat", "premium"])
df["segment"] = np.where(df["qty"] >= 3, "ulgurji", "chakana")
 
df = df.assign(revenue_k=lambda x: x["revenue"] / 1000)     # zanjir uchun qulay

`apply`dan qoching (iloji boricha)#

# ❌ Qatorma-qator Python funksiyasi — sekin
df["rev"] = df.apply(lambda row: row["price"] * row["qty"], axis=1)    # ~5.4 ms
 
# ✅ Vektorlashgan
df["rev"] = df["price"] * df["qty"]                                    # ~0.07 ms — ~75× tez

1000 qatorda farq sezilmaydi; 10 million qatorda — soniyalar va daqiqalar. apply faqat chindan vektorlashtirib bo'lmaydigan murakkab mantiq uchun (masalan, har bir matnga tokenizer). iterrows() bilan sikl — eng sekin variant; deyarli hech qachon kerak emas.

`.str` va `.dt` aksessorlari#

df["city"].str.lower()
df["city"].str.contains("kent")                     # mask
df["text"].str.len()
df["text"].str.replace(r"\s+", " ", regex=True)     # regex (18-bob)
df["email"].str.extract(r"@([\w.]+)")               # guruhni ajratib olish
 
df["date"].dt.year, df["date"].dt.day_name(), df["date"].dt.to_period("M")
df["date"] - pd.Timestamp("2026-01-01")             # Timedelta

7. Yetishmayotgan qiymatlar#

df.isna().sum()                       # har ustunda nechta
df["rating"].isna().mean()            # ulushi: 0.148
 
df.dropna(subset=["rating"])          # rating yo'q qatorlarni olib tashlash
df["rating"].fillna(df["rating"].median())     # mediana bilan to'ldirish
df["city"].fillna("noma'lum")
df["rating_missing"] = df["rating"].isna()     # "yo'qlik"ning o'zi ham ma'lumot bo'lishi mumkin!

8. Saralash, dublikatlar, tiplar#

df.sort_values("price", ascending=False)
df.sort_values(["city", "price"], ascending=[True, False])
df.nlargest(3, "revenue")
df.duplicated().sum()                           # to'liq dublikat qatorlar
df.drop_duplicates(subset=["order_id"], keep="last")
df.drop(columns=["city_upper"])
df.rename(columns={"qty": "quantity"})
 
df["category"] = df["category"].astype("category")    # takrorlanuvchi satrlar → xotira ~40× kam
df["order_id"] = df["order_id"].astype("int32")
pd.to_numeric(df["x"], errors="coerce")               # aylantirib bo'lmaydiganlar → NaN

9. `groupby` — "ajrat, qo'lla, birlashtir"#

Pandas'ning eng kuchli vositasi. SQL'dagi GROUP BY bilan bir xil g'oya:

Ajratish (split)

qatorlarni city bo'yicha guruhlarga

Qo'llash (apply)

har guruhga: count, sum, mean...

Birlashtirish (combine)

natijalar bitta jadvalga

df.groupby("city")["revenue"].sum()
 
(df.groupby("city")["revenue"]
   .agg(["count", "sum", "mean"])
   .sort_values("sum", ascending=False))

           count          sum      mean
city
Toshkent     517  278809000.0  539282.0
Samarqand    192  101110000.0  526615.0
Andijon      152   91089000.0  599270.0
Buxoro       139   86992000.0  625842.0

Qiziq: Toshkent umumiy tushumda birinchi, lekin o'rtacha buyurtma bo'yicha — oxirgi. Bunday "ikki qatlamli" xulosalar groupby'siz ko'rinmaydi.

Nomlangan agregatsiya — eng toza usul:

summary = df.groupby("city", as_index=False).agg(
    orders=("order_id", "count"),
    revenue=("revenue", "sum"),
    avg_rating=("rating", "mean"),       # NaN avtomatik e'tiborsiz qoldiriladi
)

Bir nechta kalit va unstack:

df.groupby(["city", "category"])["revenue"].sum().unstack()

Guruh ichida hisoblash: `transform`#

# Har buyurtma narxining o'z shahri o'rtachasiga nisbati
df["price_vs_city"] = df["price"] / df.groupby("city")["price"].transform("mean")

agg — har guruhdan bitta qiymat (jadval kichrayadi); transform — har qatorga guruh qiymati (jadval hajmi saqlanadi).

10. `pivot_table`, `merge`, `concat`#

df.pivot_table(index="category", columns="city", values="rating", aggfunc="mean").round(2)

city         Andijon  Buxoro  Samarqand  Toshkent
category
elektronika     3.95    3.75       3.55      3.67
kitob           3.67    4.15       3.91      3.86
kiyim           4.00    3.72       4.01      3.84

`merge` — jadvallarni birlashtirish (SQL JOIN)#

cities = pd.DataFrame({
    "city": ["Toshkent", "Samarqand", "Buxoro", "Namangan"],
    "region": ["poytaxt", "viloyat", "viloyat", "viloyat"],
})
 
merged = df.merge(cities, on="city", how="left", validate="many_to_one")
merged["region"].isna().sum()     # 152 — Andijon jadvalda yo'q edi

`how`	Natija
`inner`	Faqat ikkalasida bor kalitlar
`left`	Chapdagi hamma qatorlar; o'ngda yo'q bo'lsa — NaN
`right`	O'ngdagi hamma qatorlar
`outer`	Hammasi

pd.concat([df1, df2]) — jadvallarni ustma-ust (qatorlar bo'yicha) ulash, masalan, bir necha oylik fayllarni birlashtirish.

11. Vaqt qatorlari#

monthly = df.set_index("date").resample("ME")["revenue"].sum()   # oy oxiri bo'yicha

date
2026-01-31    57997000.0
2026-02-28    55155000.0
2026-03-31    67173000.0

monthly.pct_change()                  # oydan oyga o'zgarish
monthly.rolling(3).mean()             # 3 oylik sirpanuvchi o'rtacha
df.set_index("date").sort_index().loc["2026-03"]    # mart oyidagi buyurtmalar

12. Copy-on-Write va zanjirli tayinlash#

Pandas 3'da har bir amal natijasi mustaqil nusxa kabi ishlaydi (ichkarida esa kerak bo'lgunicha nusxalanmaydi — tejamkor). Oqibati: "zanjirli tayinlash" hech qachon asl jadvalni o'zgartirmaydi:

df[df["city"] == "Toshkent"]["price"] = 0     # ❌ ChainedAssignmentError — hech narsa o'zgarmaydi
df.loc[df["city"] == "Toshkent", "price"] = 0 # ✅ bitta qadamda — ishlaydi

Qoida: o'zgartirish — har doim bitta .loc[qator, ustun] = qiymat. Filtrlangan qismni alohida o'zgartirmoqchi bo'lsangiz — uni aniq nusxa sifatida oling va o'z nomi bilan ishlang:

tashkent = df[df["city"] == "Toshkent"].copy()
tashkent["discount"] = 0.1          # faqat tashkent jadvaliga ta'sir qiladi

13. Metod zanjiri — o'qiladigan pipeline#

Ko'p oraliq o'zgaruvchilar o'rniga, qadamlarni zanjir qiling:

report = (
    pd.read_csv("orders.csv", parse_dates=["date"])
    .assign(revenue=lambda d: d["price"] * d["qty"],
            month=lambda d: d["date"].dt.to_period("M"))
    .query("qty > 0")
    .groupby(["month", "category"], as_index=False)
    .agg(revenue=("revenue", "sum"), orders=("order_id", "count"))
    .sort_values(["month", "revenue"], ascending=[True, False])
)

Har qadam bitta qator, yuqoridan pastga o'qiladi, oraliq holat yo'q — Jupyter'da hujayralarni turli tartibda ishlatib, chalkashib ketish xavfi kamayadi.

14. Amaliyot: LLM eval natijalarini tahlil qilish#

AI muhandisining tipik vazifasi: bir nechta model/prompt variantlarining eval natijalarini solishtirish.

import numpy as np
import pandas as pd
 
rng = np.random.default_rng(1)
n = 600
evals = pd.DataFrame({
    "variant": rng.choice(["prompt_v1", "prompt_v2", "prompt_v3"], n),
    "task": rng.choice(["tasnif", "xulosa", "tarjima"], n),
    "correct": rng.random(n) < 0.8,
    "input_tokens": rng.integers(200, 2000, n),
    "output_tokens": rng.integers(20, 400, n),
    "latency_s": rng.gamma(2.0, 0.8, n).round(2),
})
 
PRICE_IN, PRICE_OUT = 4.0, 20.0       # $ / 1M token
evals["cost_usd"] = (evals["input_tokens"] * PRICE_IN + evals["output_tokens"] * PRICE_OUT) / 1e6
 
summary = (
    evals.groupby("variant")
    .agg(
        n=("correct", "size"),
        accuracy=("correct", "mean"),
        p50_latency=("latency_s", "median"),
        p95_latency=("latency_s", lambda s: s.quantile(0.95)),
        cost_per_1k=("cost_usd", lambda s: s.mean() * 1000),
    )
    .sort_values("accuracy", ascending=False)
    .round(3)
)
print(summary)
 
by_task = evals.pivot_table(index="task", columns="variant", values="correct", aggfunc="mean").round(3)
print(by_task)

Bitta jadvalda: aniqlik, kechikish (median va p95 — o'rtacha emas!), 1000 so'rov narxi. pivot_table esa qaysi variant qaysi vazifada kuchliroq ekanini ko'rsatadi. Real hayotda shu tahlil "qaysi promptni production'ga chiqaramiz" degan qarorning asosi bo'ladi.

Nima noto'g'ri ketishi mumkin#

Xato	Belgisi	Tuzatish
`df[mask]["col"] = x`	`ChainedAssignmentError`, o'zgarish yo'q	`df.loc[mask, "col"] = x`
`and`/`or` filtrlarda	`truth value of a Series is ambiguous`	`&`, `\|`, `~` va qavslar
`apply(axis=1)` / `iterrows`	Katta datada juda sekin	Vektorlashgan ustun amallari
`merge`dan keyin qatorlar ko'paydi	Noto'g'ri yig'indilar	`validate=`, qatorlar sonini tekshiring
Sana satr bo'lib qoldi	`.dt` ishlamaydi, saralash noto'g'ri	`parse_dates=` yoki `pd.to_datetime`
`int` ustun `float` bo'lib qoldi	NaN tufayli	`Int64` (nullable) tipi yoki NaN'ni hal qiling
Test ma'lumotidan to'ldirish	Data leakage	Statistikani faqat train'dan
`df.loc[0:5]` 6 qator qaytardi	`loc`da oxiri kiradi	Pozitsiya uchun `iloc`
CSV'ga `index=False`siz yozish	Keraksiz `Unnamed: 0` ustuni	`index=False`
Katta CSV xotiraga sig'maydi	`MemoryError`	`usecols`, `dtype`, `chunksize`, Parquet

Amaliy topshiriq

1. EDA

orders.csv bo'yicha javob bering: (a) qaysi kategoriya eng ko'p tushum keltiradi va uning ulushi necha foiz; (b) haftaning qaysi kunida buyurtmalar eng ko'p; (c) reyting qoldirmagan mijozlar ulushi shaharlar bo'yicha farq qiladimi; (d) eng qimmat 1% buyurtmalar umumiy tushumning necha foizini tashkil qiladi.

Maslahatni ko'rsatish

Har savol — bitta groupby yoki value_counts. Hafta kuni: df['date'].dt.day_name(). Ulush: value_counts(normalize=True).

Amaliy topshiriq

2. Tozalash funksiyasi

Ataylab "iflos" dataset yarating: dublikatlar, manfiy narxlar, katta-kichik harf va bo'sh joylari turlicha shahar nomlari (" toshkent", "TOSHKENT"), satr ko'rinishidagi sanalar. Hammasini tuzatadigan clean(df) funksiyasini metod zanjiri bilan yozing va pytest bilan test qiling.

Maslahatni ko'rsatish

def clean(df) -> pd.DataFrame: return (df.drop_duplicates(...).assign(...).query(...)); har qadamdan keyin qatorlar sonini logga yozing.

Amaliy topshiriq

3. Kogorta tahlili

Datasetga customer_id ustunini qo'shing (200 ta mijoz, takroriy xaridlar bilan). Har mijozning birinchi xarid oyini aniqlang (transform) va kogorta jadvalini quring: qator — birinchi oy, ustun — "necha oydan keyin", qiymat — qaytgan mijozlar soni.

Maslahatni ko'rsatish

Har mijozning birinchi oyi: df.groupby('customer_id')['date'].transform('min').dt.to_period('M'); keyin pivot_table(index=cohort, columns=months_since, values=customer_id, aggfunc='nunique').

Amaliy topshiriq

4. Eval hisobotini kengaytiring

LLM eval misoliga: (a) har variant uchun accuracy'ning 95% ishonch oralig'ini (bootstrap yoki p ± 1.96·sqrt(p(1-p)/n)) qo'shing; (b) narx va aniqlik bo'yicha "Pareto-optimal" variantlarni aniqlang; (c) natijani Markdown jadval sifatida chiqaring (df.to_markdown(), uv add tabulate).

Yodda saqlang

01DataFrame — umumiy indeksli Series'lar jadvali; ichida NumPy/Arrow massivlari.
02Har doim boshlang: head, shape, info, describe, value_counts, isna().sum().
03loc — yorliq bo'yicha (oxiri kiradi), iloc — pozitsiya bo'yicha; filtr: &, |, ~ va qavslar.
04Vektorlashgan ustun amallari; apply(axis=1) va iterrows — oxirgi chora.
05groupby + agg (nomlangan) — xulosa; transform — guruh qiymati har qatorga.
06merge'dan keyin qatorlar sonini tekshiring, validate= ishlating.
07Pandas 3: Copy-on-Write — o'zgartirish faqat df.loc[mask, col] = value.
08Metod zanjiri — o'qiladigan, takrorlanuvchan pipeline; katta data uchun Parquet.

Pandas: jadval ma'lumotlari bilan ishlash

1. Mashq uchun dataset#

2. Series va DataFrame#

3. O'qish va saqlash#

4. Ma'lumotni birinchi ko'rish#

5. Tanlash: `[]`, `loc`, `iloc`#

Filtrlash#

6. Yangi ustunlar va vektorlashtirish#

`apply`dan qoching (iloji boricha)#

`.str` va `.dt` aksessorlari#

7. Yetishmayotgan qiymatlar#

8. Saralash, dublikatlar, tiplar#

9. `groupby` — "ajrat, qo'lla, birlashtir"#

Guruh ichida hisoblash: `transform`#

10. `pivot_table`, `merge`, `concat`#

`merge` — jadvallarni birlashtirish (SQL JOIN)#

11. Vaqt qatorlari#

12. Copy-on-Write va zanjirli tayinlash#

13. Metod zanjiri — o'qiladigan pipeline#

14. Amaliyot: LLM eval natijalarini tahlil qilish#

Nima noto'g'ri ketishi mumkin#

1. EDA

2. Tozalash funksiyasi

3. Kogorta tahlili

4. Eval hisobotini kengaytiring

Test va amaliy masalalar

Bob bo'yicha test

Amaliy masalalar

01Shaharlar bo'yicha hisobot

02Tozalash pipeline'i

03Eval natijalari jadvali

1. Mashq uchun dataset#

2. Series va DataFrame#

3. O'qish va saqlash#

4. Ma'lumotni birinchi ko'rish#

5. Tanlash: [], loc, iloc#

Filtrlash#

6. Yangi ustunlar va vektorlashtirish#

applydan qoching (iloji boricha)#

.str va .dt aksessorlari#

7. Yetishmayotgan qiymatlar#

8. Saralash, dublikatlar, tiplar#

9. groupby — "ajrat, qo'lla, birlashtir"#

Guruh ichida hisoblash: transform#

10. pivot_table, merge, concat#

merge — jadvallarni birlashtirish (SQL JOIN)#

11. Vaqt qatorlari#

12. Copy-on-Write va zanjirli tayinlash#

13. Metod zanjiri — o'qiladigan pipeline#

14. Amaliyot: LLM eval natijalarini tahlil qilish#

Nima noto'g'ri ketishi mumkin#

1. EDA

2. Tozalash funksiyasi

3. Kogorta tahlili

4. Eval hisobotini kengaytiring

Test va amaliy masalalar

Bob bo'yicha test

Amaliy masalalar

5. Tanlash: `[]`, `loc`, `iloc`#

`apply`dan qoching (iloji boricha)#

`.str` va `.dt` aksessorlari#

9. `groupby` — "ajrat, qo'lla, birlashtir"#

Guruh ichida hisoblash: `transform`#

10. `pivot_table`, `merge`, `concat`#

`merge` — jadvallarni birlashtirish (SQL JOIN)#