Commit 472f2d2b authored by O'Reilly Media, Inc.'s avatar O'Reilly Media, Inc.
Browse files

Initial commit

parents
Loading
Loading
Loading
Loading
+7 −0
Changes for Natural Language Text Processing with Python - Working Files/Chapter 1/No Work Files.rtf: 7 added lines, 0 removed lines.
Original line number Diff line number Diff line
{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\margl1440\margr1440\vieww9000\viewh8400\viewkind0
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

\f0\fs24 \cf0 No work files for this lesson...}
 No newline at end of file
+7 −0
Changes for Natural Language Text Processing with Python - Working Files/Chapter 10/No Work Files.rtf: 7 added lines, 0 removed lines.
Original line number Diff line number Diff line
{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\margl1440\margr1440\vieww9000\viewh8400\viewkind0
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

\f0\fs24 \cf0 No work files for this lesson...}
 No newline at end of file
+127 −0
Changes for Natural Language Text Processing with Python - Working Files/Chapter 2/1 Unicode Python 2.ipynb: 127 added lines, 0 removed lines.
Original line number Diff line number Diff line
%% Cell type:code id: tags:

``` python
# -*- coding: utf-8 -*-
# In Python 2 you often need the line above at the top of your file
# so you can see Unicode characters
```

%% Cell type:code id: tags:

``` python
a = 'niño'
print(a)
```

%% Output

    niño

%% Cell type:code id: tags:

``` python
print(type(a))
```

%% Output

    <type 'str'>

%% Cell type:code id: tags:

``` python
# Use u'bob' to create unicode, 'bob' is str (ascii)
b = u'bob'
print(b)
print(type(b))
```

%% Output

    bob
    <type 'unicode'>

%% Cell type:code id: tags:

``` python
# Length of Unicode is the number of characters, but length of str is the number of bytes
# format command just makes things easier
print("Length of unicode: {}".format(len(u'à')))
print("Length of str (bytes): {}".format(len('à')))
```

%% Output

    Length of unicode: 1
    Length of str (bytes): 2

%% Cell type:code id: tags:

``` python
# A lot of times you will get text from some unknown encoding.
# UTF-8 is the most common representation.
# If you need to catch errros, you can remove errors='ignore'
def convert_unicode(text):
    if isinstance(text,str):
        return text.decode('utf-8',errors='ignore')
    else:
        return text
in_str = 'à'
print(type(in_str))
in_unicode = convert_unicode(in_str)
print(type(in_unicode))
```

%% Output

    <type 'str'>
    <type 'unicode'>

%% Cell type:code id: tags:

``` python
# But to write to a file, it must be encoded back to bytes
# --> Gives UnicodeEncodeError
a = u'à'
print(type(a))
with open('tmp.txt','w') as f:
    f.write(a)
print('done')
```

%% Output

    <type 'unicode'>

    ---------------------------------------------------------------------------
    UnicodeEncodeError                        Traceback (most recent call last)
    <ipython-input-7-808c054e9f11> in <module>()
          4 print(type(a))
          5 with open('tmp.txt','w') as f:
    ----> 6     f.write(a)
          7 print('done')
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 0: ordinal not in range(128)

%% Cell type:code id: tags:

``` python
# Instead encode it like this
a = u'à'
print(type(a))
with open('tmp.txt','w') as f:
    a = a.encode('utf-8',errors='ignore')
    f.write(a)
print(type(a))
print('done')
```

%% Output

    <type 'unicode'>
    <type 'str'>
    done

%% Cell type:code id: tags:

``` python
```
+43 −0
Changes for Natural Language Text Processing with Python - Working Files/Chapter 2/2 Unicode Python 3.ipynb: 43 added lines, 0 removed lines.
Original line number Diff line number Diff line
%% Cell type:code id: tags:

``` python
# In Python 3 str is Unicode
a = 'niño'
print(type(a))
```

%% Output

    <class 'str'>

%% Cell type:code id: tags:

``` python
# It gives the length as one character
x = 'à'
print(len(x))
```

%% Output

    1

%% Cell type:code id: tags:

``` python
# Unicode is standard in Python 3, making life easier in that regard
print(type(x))
with open('tmp.txt','w') as f:
    f.write(x)
print('done')
```

%% Output

    <class 'str'>
    done

%% Cell type:code id: tags:

``` python
```
+192 −0
Changes for Natural Language Text Processing with Python - Working Files/Chapter 2/string processing.ipynb: 192 added lines, 0 removed lines.
Original line number Diff line number Diff line
%% Cell type:code id: tags:

``` python
# You can add strings
a = "chicken" + "man"
print(a)
```

%% Output

    chickenman

%% Cell type:code id: tags:

``` python
# A string is a sequence of characters starting with 0
print(a[1])
print(a[2:])
print(a[2:-1])
```

%% Output

    h
    ickenman
    ickenma

%% Cell type:code id: tags:

``` python
# you can check if a string is in another string
print("hi" in a)
print("zi" in a)
print("HI" in a)
print("HI".lower() in a)
```

%% Output

    True
    False
    False
    True

%% Cell type:code id: tags:

``` python
# you can check starting and ending
print(a.startswith("ch"))
print(a.endswith("man"))
```

%% Output

    True
    True

%% Cell type:code id: tags:

``` python
# but strings cannot be changed (they are immutable)
print(a)
print(a[3])
```

%% Output

    chickenman
    c

%% Cell type:code id: tags:

``` python
# but strings cannot be changed (they are immutable)
# more immutable, you can't replace the second 'c' with 'X'
a[3] = 'X'
```

%% Output

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-6-643a1d946f27> in <module>()
          1 # but strings cannot be changed (they are immutable)
          2 # more immutable, you can't replace the second 'c' with 'X'
    ----> 3 a[3] = 'X'

    TypeError: 'str' object does not support item assignment

%% Cell type:code id: tags:

``` python
# if you want to change a part of the string, use +
new_string = a[:3] + 'X' + a[4:]
print(new_string)
print(a)
```

%% Output

    chiXkenman
    chickenman

%% Cell type:code id: tags:

``` python
# you can split a string based on a character
word_list = "I love to drive cars.".split(' ')
print(word_list)
```

%% Output

    ['I', 'love', 'to', 'drive', 'cars.']

%% Cell type:code id: tags:

``` python
# you can strip off line breaks ('\n')
return_at_end = "Hi there.\n"
print(return_at_end)
print("Next line.")
print(return_at_end.strip())
print("Next line.")
```

%% Output

    Hi there.
    
    Next line.
    Hi there.
    Next line.

%% Cell type:code id: tags:

``` python
# you can replace characters in a string
new_string = "I love tacos.".replace('love','enjoy')
print(new_string)
```

%% Output

    I enjoy tacos.

%% Cell type:code id: tags:

``` python
# strings can be ' or " or ''' or """
a = 'bob'
b = "tom"
c = '''frank'''
d = """frank"""
```

%% Cell type:code id: tags:

``` python
# need to escape
print('I can't believe it.')
```

%% Output

      File "<ipython-input-12-518b6ec77231>", line 2
        print('I can't believe it.')
                     ^
    SyntaxError: invalid syntax

%% Cell type:code id: tags:

``` python
# can escape or put it inside other kind
print("I can't believe it.")
print('I can\'t believe it.')
```

%% Output

    I can't believe it.
    I can't believe it.

%% Cell type:code id: tags:

``` python
```

%% Cell type:code id: tags:

``` python
```
Loading