top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Python Cramers' V Handling Missing Values & Date Fields

0 votes
370 views

I'm working on association analysis on my dataset that consist majorly of categorical features and I'm using Cramers' V and Theils U statistical measures for showcasing the association metrics.

I have 2 questions related to the same :

If there are some missing values in some of the columns in my dataset, how should I handle them while calculation Cramers' V and theils u metrics? Shall I replace the missing value with some dummy value?
Note: I'm using python's python library for the calculation of both the metrics.

dython.nominal.cramers_v(data[field1],data[field2]) and dython.nominal.theils_u(data[field1],data[field2])

If I have a column name like "Task Creation Date" that consists of DateTime values. How can I include this field as part of my association analysis? Does Cramers' V and Theils U consider date values as input ? or some conversion is required?

Any help would be much appreciated.

posted Nov 17, 2020 by Jainsaniya

Looking for an answer?  Promote on:
Facebook Share Button Twitter Share Button LinkedIn Share Button

Similar Questions
0 votes

On building Python 2.7.5 I got the following message:

Python build finished, but the necessary bits to build these modules were not found:
dl imageop linuxaudiodev 
spwd sunaudiodev 
To find the necessary bits, look in setup.py in detect_modules()  for the module's name.

It carried on with the installation OK, but I don't understand the last sentence in the message. How can I find out exactly what modules are missing, and what I need to do to make sure they are built next time?

+2 votes

Starting on any day/date, I would like to create a one year list, by week (start date could be any day of week). Having a numerical week index in front of date, ie 1-52, would be a bonus.
ie, 1. 6/4/2013
2. 6/11/2013
3. 6/18/2013....etc to # 52.

And to save that result to a file. Moving from 2.7 to 3.3

+1 vote

I have an object (a variable) name, which gets its value from a PostgreSQL database via a SELECT statement, an it sometimes has special characters as ß, ä, ö...

Then I would like to insert that value into a table in a SQLite database. So I make a cursor cur on the table and prepare a SQL statement like this:

sql = 'insert into tbl values(?)'
cur.execute(sql, (name,))

That ends up with the exception, for example,

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6:  ordinal not in range(128)

The "position 6" is exactly the position of the special character, ß in this case. What to do?

+1 vote

Why the maximum and minimum exp values are 1024 and -1021?:

 >>> sys.float_info
sys.float_info(max=1.7976931348623157e+308, max_exp=1024, 
max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, 
min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, 
radix=2, rounds=1)

The values (in double precision) 0 and 2047 are reserved for zero, infinity and NaN (in combination with the fraction), so I was expecting -1022 and 1023...

0 votes

I am trying to use mitmproxy behind a company proxy that requires a user/password login.

The setup is: Local PC's browser -> mitmproxy (on local PC) -> company proxy -> internet.

Based on this SO thread, this is how you use mitmproxy within a Python program. This example works fine when there's no proxy.

from mitmproxy.options import Options
from mitmproxy.proxy.config import ProxyConfig
from mitmproxy.proxy.server import ProxyServer
from mitmproxy.tools.dump import DumpMaster

class Addon(object):
    def __init__(self):
        pass

    def request(self, flow):
        # examine request here 
        pass

    def response(self, flow):
        # examine response here
        pass


if __name__ == "__main__":

    options = Options(listen_host='0.0.0.0', listen_port=8080, http2=True)
    m = DumpMaster(options, with_termlog=False, with_dumper=False)
    config = ProxyConfig(options)

    m.server = ProxyServer(config)
    m.addons.add(Addon())

    try:
        print('starting mitmproxy')
        m.run()
    except KeyboardInterrupt:
        m.shutdown()

Assuming the company proxy is at IP "1.2.3.4" port 3128 and requires a login USER and PASSWORD, how can I change this script to have mitproxy use that proxy instead of going to the internet directly?

Addition info: I am not using mitmdump with the script-parameter to run this script. The goal is to run this from Python 3.8 with a pip-installed mitmproxy

...