CounterExample_For_Least_Square_Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is it possible to indicate that in a least square model: \n",
    "$$\n",
    "\\begin{align}\n",
    "y = \\beta_0 + \\sum_{i = 1}^{p-1}\\beta_i \\implies \\widehat{\\beta_i} \\perp\\!\\!\\!\\!\\perp \\widehat{\\beta_j} \\iff \\sum_{k = 1}^n x_{i, k} x_{j, k} = 0, \\quad \\text{and} ~~i \\ne j. \n",
    "\\end{align}\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We know that in a least square model the vector of coefficients estimated by below relation: \n",
    "$$\n",
    "\\begin{aligned}\n",
    "& \\widehat{\\beta} = (X'X)^{-1} X' y \\implies \\text{Var}(\\widehat{\\beta}) = \\sigma^2 \\times (X'X)^{-1} \\implies \\\\\n",
    "& \\widehat{\\beta} \\sim \\mathcal{N}_p (\\beta, ~~\\sigma^2 (X'X)^{-1}) \\implies \\\\\n",
    "& \\text{if}\\quad \\mathbb{V} = (X'X)^{-1} \\implies \\mathbb{Cov}(\\widehat{\\beta_i}, \\widehat{\\beta_j}) = \\sigma^2 \\times \\mathbb{V}[i, j] \\implies \\\\\n",
    "& \\widehat{\\beta_i} \\perp\\!\\!\\!\\!\\perp \\widehat{\\beta_j} \\iff \\mathbb{V}[i+1, j+1] = 0 \n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I give an Counterexample for the relationship (1)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let \n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "& x_1 = \\begin{bmatrix}\n",
    "1 \\\\ 2 \\\\ 3 \\\\ 4 \\\\ 5  \n",
    "\\end{bmatrix} \\quad x_2 = \\begin{bmatrix}\n",
    "2 \\\\ 4 \\\\ 7 \\\\ 1 \\\\ -7 \n",
    "\\end{bmatrix} \\implies  \\\\\n",
    "& X = \\begin{bmatrix} 1 & 1 & 2 \\\\ 1 & 2 & 4 \\\\ 1 & 3 & 7  \\\\ 4 & 1 & 1 \\\\ 1 & 5 & -7\n",
    "\\end{bmatrix} \\\\\n",
    "& \\implies n = 5, ~~ p = 3, \\\\\n",
    "& X' X = \\begin{bmatrix} \n",
    "5 & 15 & 7 \\\\ \n",
    "15 & 55 & 0 \\\\\n",
    "7 & 0 & 119\n",
    "\\end{bmatrix}   \\implies \\sum_{k = 1}^5 x_{1, k} x_{2, k} = 0 \\\\ \n",
    "& \\mathbb{V} = (X'X)^{-1} = \\begin{bmatrix} \n",
    "\\frac{187}{93} & -\\frac{17}{31} & -\\frac{11}{93} \\\\\n",
    "-\\frac{17}{31} & \\frac{26}{155} & \\frac{1}{31} \\\\\n",
    "-\\frac{11}{93} & \\frac{1}{31} & \\frac{10}{651} \n",
    "\\end{bmatrix} \\implies \\\\\n",
    "& \\mathbb{Cov}(\\widehat{\\beta_1}, ~~\\widehat{\\beta_2}) = \\sigma^2 \\times \\mathbb{V}[2, 3] = \\sigma^2 \\times \\frac{1}{31} \\ne 0\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*** \n",
    "***\n",
    "\n",
    "\n",
    "## Using Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sympy import * \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}1 & 1 & 2\\\\1 & 2 & 4\\\\1 & 3 & 7\\\\1 & 4 & 1\\\\1 & 5 & -7\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[1, 1,  2],\n",
       "[1, 2,  4],\n",
       "[1, 3,  7],\n",
       "[1, 4,  1],\n",
       "[1, 5, -7]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1 = [1, 2, 3, 4, 5]\n",
    "x2 = [2, 4, 7, 1, -7]\n",
    "\n",
    "\n",
    "X = Matrix(np.array([1, 1, 1, 1, 1, 1, 2, 3, 4, 5, 2, 4, 7, 1, -7]).reshape(3, 5).T)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}5 & 15 & 7\\\\15 & 55 & 0\\\\7 & 0 & 119\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[ 5, 15,   7],\n",
       "[15, 55,   0],\n",
       "[ 7,  0, 119]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "XX = X.T @ X \n",
    "XX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}\\frac{187}{93} & - \\frac{17}{31} & - \\frac{11}{93}\\\\- \\frac{17}{31} & \\frac{26}{155} & \\frac{1}{31}\\\\- \\frac{11}{93} & \\frac{1}{31} & \\frac{10}{651}\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[187/93, -17/31, -11/93],\n",
       "[-17/31, 26/155,   1/31],\n",
       "[-11/93,   1/31, 10/651]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xxinverse = XX.inv()\n",
    "xxinverse"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}