torchtext.data 的 Field, RawField

今天试图更改open-nmt代码时,在preprocess阶段发现一处代码:

    fields = inputters.get_fields( 
        opt.data_type,
        src_nfeats,
        t >_nfeats,
        dynamic_dict=opt.dynamic_dict,
        src_truncate=opt.src_seq_len >h_trunc,
        t >_truncate=opt.t >_seq_len >h_trunc)

而fields的各个组成部件为不同的类型

    fields["t >"] = fields_getters["text"](**t >_field_kwargs)          # TextMu< iField 

    indices = Field(use_vocab=False, dtype=torch.long, sequential=False) # 
    fields["indices"] = indices

    if dynamic_dict:
        src_map = Field(
            use_vocab=False, dtype=torch.float,
            postprocessing=make_src, sequential=False)
        fields["src_map"] = src_map

        src_ex_vocab = RawField()
        fields["src_ex_vocab"] = src_ex_vocab

        align = Field(
            use_vocab=False, dtype=torch.long,
            postprocessing=make_t >, sequential=False)
        fields["alignment"] = align

可以看到有的是TextMu< iField,有的是Field, RawField

这是为什么呢?有什么区别呢?

 

打开torchtext官方文档:https://torchtext.readthedocs.io/en/latest/data.html#field

查找到如下内容:

Field

class torchtext.data.Field(sequential=True, use_vocab=True, init_token=None, eos_token=None, fix_len >h=None, dtype=torch.int64, preprocessing=None, postprocessing=None, lower=False, tokenize=None, tokenizer_language='en', include_len >hs=False, batch_first=False, pad_token='pad', unk_token='unk', pad_first=False, truncate_first=False, stop_words=None, is_target=False)
Defines a datatype together with instructions for converting to Tensor.

Field class models common text processing datatypes that can be represented by tensors. It holds a Vocab object that defines the set of possible values for elements of the field and their corresponding numerical representations. The Field object also holds other parameters relating to how a datatype should be numericalized, such as a tokenization method and the kind of Tensor that should be produced.

定义一个数据类型和转换成张量的指令。

字段类模型可由张量表示的常见文本处理数据类型。
它持有一个Vocab对象,该对象定义字段元素的可能值集及其相应的数值表示。
Field对象还包含与数据类型如何数字化相关的其他参数,比如记号化方法和应该生成的张量类型。

If a Field is shared between two columns in a dataset (e.g., question and answer in a QA dataset), then they will have a shared vocabulary.

Variables:变量详情见上述网址

 

RawField

class torchtext.data.RawField(preprocessing=None, postprocessing=None, is_target=False)
Defines a general datatype.

Every dataset consists of one or more types of data. For instance, a text classification dataset contains sentences and their classes, while a machine translation dataset contains paired examples of text in two languages. Each of these types of data is represented by a RawField object. A RawField object does not assume any property of the data type and it holds parameters relating to how a datatype should be processed.

定义一般数据类型。

每个数据集都包含一种或多种类型的数据。
例如,文本分类数据集包含句子及其类,而机器翻译数据集包含两种语言的成对文本示例。
这些类型的数据都由一个RawField对象表示。
RawField对象不假设数据类型的任何属性,它持有与处理数据类型相关的参数。

Variables:

 

 

 

 

最新回复(0)
/jishuZZDkdJ2V6dq0VaK9J7rbRft9R1J1Mqys5KH9fA_3D_3D4795239
8 简首页